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ABSTRACT 

An evaluation of the fifth cycle of operation of the 
Massachusetts Workplace Education Initiative had several purposes: 
(1) to identify the indicators of quality in workplace education; (2) 
to begin to design an evaluation system for each program; and (3) to 
assess the indicators of quality for workplace education programs 
against the national indicators of quality in adult education. Five 
programs in the state were included in the evaluation, which 
consisted of onsite, team-based evaluations of indicators of quality. 
Phases of the evaluation process were as follows: introduction, team 
meetings, follow-ups to meetings, drafting of an evaluation 
instrument, review of the evaluation process, and synthesis of 
results across the programs. Among the quality outcomes desired were 
improved productivity, improved product quality, a flexible work 
force, improved communication in English, improved employee 
involvement, and career growth. Methods of achieving quality outcomes 
were identified: recruiting participants, fostering self-esteem, 
interweaving program and organization goals, and identifying barriers 
to and incentives for employee participation. It was concluded that 
indicators of quality for workplace education programs should be 
reviewed with critical attention to the organizational context of 
these programs and that evaluation should be an ongoing, integrated 
Sanction. Recommendations were made to implement the results of the 
Cudy throughout the programs, (Contains 15 references.) (KC) 
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Introduction: The Evolution of the Planning and Evaluation 
Team as A Way to Evaluate Quality in Workplace Education 



In March of 1992, the Bureau of Adult Education in the Massachusetts 
Department of Education contracted with Evaluation Research, an independent 
consulting company, to conduct an evaluation of the fifth cycle of operation of 
the Massachusetts Workplace Education Initiative (MWEI). The broad goal of 
the MWEI Cycle 5 evaluation was to build on the conclusions and 
recommendations of the Cycle 4 MWEI Evaluation by refining a self-evaluation 
process for workplace education programs which uses evaluation teams at 
each participating program site. More specific goals included: 

(1) help evaluation teams to identify the "indicators of quality in 
workplace education" that are most applicable to their program. Indicators of 
quality include both program outcomes and program processes; 

(2) begin to design an evaluation system in each program based on 
documenting and/or measuring these indicators of quality.and, when possible, 
to draft instruments, collect data, and evaluate programs according to these 
indicators; 

(3) assess the indicators of quality for workplace education programs 
identified by all the participating programs against the national indicators of 
quality in adult education. 

Participatin g Programs 

The programs which participated in the MWEI Cycle 5 evaluation are: 

• James River/Amarin, the Montachusett Opportunity Council, and the 
Montachusett Private Industry Council, Leominster, MA. 

• Michael J. Neville Manor Nursing Home and the Adult Learning Center, 
Cambridge, MA. 

• Nypro, Inc. and the Clinton Adult Learning Center, Clinton, MA. 

• The Norton Company, Bonded Abrasives Division, and the Workplace 
Education Program, Center for Life-Long Learning, Quinsigamond Community 
College, Worcester, MA. 

• International Ladies' Garment Workers Union (I.L.G.W.U.), the Asian American 
Civic Association, and Eleven Small Garment Shops in Boston, MA. 

Brief History of the MWEI Evaluations 

Since it began in 1985, the MWEI has had an unusually strong 
commitment to program evaluation and to using the information which the 
evaluations gathered to inform program development and public policy. This 
commitment to evaluation by the State partners translated into freedom for the 
evaluators to explore the evaluation strategies that would best suit the multi- 
level needs of the State and local partnerships. 
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• In Cycle 1 of the MWEI, the evaluators (Bruce Astrein, Laura Sperazi 
and Adria Steinberg) conducted intensive two-day site visits with all the players 
involved with the six programs that composed the first program cohort. 

• In Cycle 2, the evaluators (Laura Sperazi and Bruce Astrein) 
continued to conduct intensive site visits with the programs in the second cohort 
- now increased to a total number of seventeen. 

• In Cycle 3, the evaluators (Laura Sperazi and Paula Rayman) turned 
their attention from documenting effective principles and practices to 
documenting and measuring program outcomes. 

• In Cycle 4, the evaluators (Laura Sperazi and Paul Jurmo) worked 
with evaluation teams at each site to (1) identify the outcomes which the teams 
wanted their programs to achieve; (2) design data-gathering instruments and 
processes to document and/or measure the achievement of those outcomes 
over time; (3) analyze the data which was collected; and (4) report the 
information to appropriate audiences. 

• In Cycle 5, the evaluator (Laura Sperazi) extended the team-based 
outcome evaluation model to include concepts of both quality program 
outcomes and program processes. 



II. The Planning and Evaluation Team Method: Seven 
Phases in Cycle 5 

Background and Purpose 

The evaluation of Cycle 5 of the MWEI consisted of on-site, team- 
based evaluations of "indicators of quality" in five workplace education 
programs funded through the U.S. Department of Education, National 
Workplace Literacy Program. The members of the evaluation teams 
represented the spectrum of the workplace education partnership -- program 
participants, supervisors, unions, managers, and educators. First the evaluator 
helped each team to define what quality looks like in their program ~ both 
quality outcomes (what a program wants to achieve in the long run) and quality 
processes (what a program must have in place in order to achieve its quality 
outcomes.) Then the evaluator helped each team to think about customizing an 
evaluation system that would document and/or measure over time whether ano 
how the program is achieving those quality outcomes and processes. 

The long term goal of this team-based evaluation method is to make 
evaluation as integral a part of program operations as curriculum development, 
recruitment, or any other essential program component. Team members know 
what quality looks like in their programs. They know the signs of doing 
something right -- or not. They can use what they know to develop the criteria 
against which they evaluate themselves. And they can use the information they 
gather through their customized evaluation system to continuously improve their 
program operations. 

The team-based evaluation process frequently refers to the "planning 
and evaluation spiral" to emphasize the feedback relationship between 




evaluation and strategic planning. This focus on evaluation being put to 
practical use encourages the teams to take their work seriously. It means that 
they will see the results of their work in an improved program. This process is 
continuous, spiralling upward in a continuously integrated learning pattern 
This process mirrors the processes of total quality management. It is continuous 
improvement in practice. 

Not surprisingly, evaluation teams in Cycle 5 started in different places, 
had different issues and resources to work with, and accomplished different 
types and degrees of work. All the teams identified what quality outcomes they 
wanted their programs to achieve and what their programs must have in place 
in order to achieve those outcomes. Some teams were able to move very 
quickly into developing an evaluation system with draft instruments to collect 
data on whether and how the program is achieving those outcomes. Other 
teams moved more slowly, having to grapple more intensively with program 
process issues like the mechanics of collaboration between education and 
management. 

Phases o f the Evaluation Process 

Phase 1: Introduce the Cycle 5 Evaluation to Education Staff from 
Participating Programs (March, 1992; one two-and-a-half hour meeting) 

In a meeting convened for the purpose of introducing the Cycle 5 
evaluation to the education staff from participating programs, the evaluation 
facilitator described the history of the MWEI evaluations Cycles 1 through 4; 
summarized the goals and phases of the Cycle 5 evaluation; presented a time- 
line for work to be completed; and was available to answer questions. 



Phase 2: Convene the First Team Meeting. (May - September, 1992; 
two hours or more. 

In the first team meeting, the facilitator replicated on a smaller scale for 
the full evaluation teams the presentation made to the education providers in 
their introductory meeting about the goals, history and time-line of the Cycle 5 
evaluation. In this meeting, the evaluation teams accomplished the tasks listed 
below. 

•Presentation and discussion of the philosophy of the MWEI on 
evaluation. 

•Presentation and discussion of the history of the MWEI 
evaluations. 

•Review and discussion of the evaluation's focus on quality. 

•Review and discussion of the evaluation's focus on the planning 
and evaluation spiral, and how it is consistent with the continuous 
improvement model. 
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•Facilitation of a mini-workshop where team members define 
quality outcomes and processes. 

The facilitator asks series of questions related to identifying quality outcomes - 
and processes. The facilitator asks two basic questions related to identifying 
quality outcomes and processes. These questions are: 

- What do you want your program to achieve - what are its desired long and 
short-term "quality outcomes?" 

- What quality processes must your program have in place in order to achieve 
its quality outcomes? 

• Review and discussion of the quality outcomes and processes 
which the team identified, and preparation for data collection: 
How can the team document and/or measure whether and how the 
program is achieving these quality outcomes and implementing 
these quality processes? 

• Identification of the next steps which the team needs to take in 
order to advance its evaluation. 

• The facilitator records all discussion. 



Phase 3: Follow-up to the First Team Meeting (May - September, 
1992) 

The facilitator follows-up the first team meeting in three ways: 

• Summarize the work the team accomplished in its first meeting, 
and report it back to team members. 

•Identify materials related to conducting a team based evaluation 
with a focus on quality and forward them to team members with 
the first follow-up memo. These materials include: 

- The Malcolm Baldrige National Award Guidelines. 

- Workplace Education in Context: A Chart Comparing Traditional and High 
Performance Work Organizations. 

• Distribute other evaluation-related materials and make follow-up 
telephone calls as necessary. 



Phase 4: Convene the Second Team Meeting. (June - September, 
1992; two hours or more) 

In the second team meeting, the facilitator reviews the work 
accomplished in the first team meeting and moves the team to the point of 
drafting an evaluation instrument that would serve as the first piece of an 
evaluation system. 

• Review the main issues which emerged in the first team meeting. 

• Review and prioritize quality outcomes and processes identified 
in first team meeting. 
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• Presentation of essential components of an evaluation system 
-- how to best document/measure the team's high priority outcomes and 

processes: interview, survey, checklist, observation, simulation, or other 

means. 

- whether already existing data in the organization can be used to document 
and/or measure them 

-- the process for drafting the instrument(s) 

- the process for finalizing the draft instrument(s) 

- assigning responsibility for piloting the instrument(s) 

- assigning responsibility for collecting the data 
~ assigning responsibility for analyzing the data 

- assigning responsibility for reporting the data 

• Draft an evaluation instrument Team members design a first draft of an 
evaluation instrument which will serve as the starting point of the evaluation. 

• Introduce a data-base program called Statistical Package for the 
Social Sciences (SPSS) with team members in programs where 
the number of participants, and the amount of data to be 
collected, makes this an efficient option. (Ten minutes) 



Phase 5: Follow-up to the Second Team Meeting (June - September 
1992) K 

The facilitator follows-up the second team meeting in three ways: 

• Summarize the work the team accomplished in its second 
meeting, and report it back to team members. 

• Draft or review drafts of evaluation instruments. 

• Conduct a third team meeting as needed. 



Phase 6: Teams Meet Together to Review the Evaluation Process 
and Share Information 

In late August, after four of the five teams had completed their meetings 
with the evaluation facilitator, team members from all participating programs 
convened for an information sharing meeting. Two important results of the 
meeting were: 

(1) Networking among company managers. 

(2) The presence of program participants from two teams reinforced 
everyone's understanding of how important it is to have orogram 
participants deeply involved in the evaluation process. 

Phase 7: The Evaluator Synthesizes Data on Quality Outcomes and 

Processes across Programs 

The synthesis is reported in Section III, below. 
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Hi. Quality Outcomes and Quality Process: What Teams 
Want Their Programs to Achieve, and How to Get There 
- Local and National Perspectives 

As stated in the introduction to this report, one of the goals of the Cycle 5 
MWEI evaluation is to assess the outcomes and program processes that teams 
identify as indicators of quality in workplace education across the participating 
programs. The team-based method of evaluation which is described in this 
report first and foremost respects the need for each program partnership to 
identify quality outcomes and processes in terms that pertain specifically to it. 
However, at the same time, it is important to look at these definitions across 
programs for similarities and differences that will help the field in general to 
understand what quality in workplace education looks like. 

The data are useful for at least two reasons. First, because of what they 
suggest national indicators of quality in workplace education should include 
that are different from the generic indicators established for the adult education 
field. Second, because of how they may be used in or adapted for evaluation 
team meetings as resource materials when team members are beginning the 
process of identifying quality goals and processes. 

Chart #1 synthesizes what the teams identified as their desired 
outcomes. Chart #2 summarizes what programs must have in place in order to 
achieve quality outcomes. These are indicators of quality in program process. 
The format is the same as for Chart #1. 
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Chart #1 

Outcomes: What Tea ms Want Their Programs to Achieve 
! Improved Productivity {5}* 
! Improved Product Quality {4} 

* Flexible Workforce {4} 

• Improved Communication in Enolish {4} 
; Improved Employee Involvement {4} 

! Continued Education and Training {4} 

* Career Growth {4} 

; Improved Awareness and Appreciation of Cultural Differences {3} 

! Containing Supervisors' Responsibilities {3} 

! Accurate Documentation of Work on Appropriate Forms {3} 

; Improved Safety {3} 

Improved Math Skills {2} 

• Improved Quality of Life {2} 
! Reduced scrap {2} 

• Improved scrap management {1} 

* Number of teams who identified this quality outcome. 
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Chart # 2 

Program Process! H ow to Achieve Quality Outcome 

• identify and Recruit Pr ogram Participants {5}* 
! Foster Self Esteem {$}. 

interweave Program Go als with the Goals of the Organization {4} 

! Build and Maintain Hioh Quality in the Business/Education Partnership {4} 

! identify Barriers to an d Incentives for Employee Participation {4} 

! identify Specific Vocabulary for ESL Instruction {4} 

! Retain Em ployees in the Education Program {2} 

! Other Process Indicators of Quality 

- Build community: I.LG.W.U. 

-- Resolve issues of coverage during class hours: Neville Manor 

- Re-enroll dropouts: Neville Manor 

- Positive feedback from the students: Neville Manor 

- Have a permanent setting for classes: Neville Manor 

* Number of teams who identified this quality process. 



Local Indicators of Quality in the National Content 

In "Model Indicators of Program Quality for Adult Education Programs " 
the July 1992 publication of the Office of Vocational and Adult Education, eight 
model indicators for the field of adult education are listed. It is significant' to note 
that, while the programs which participated in this evaluation comply with these 
eight model indicators of quality, evaluation teams identified only three of the 
eight indicators as priority indicators for their programs. The three indicators 
are: 



• #2 Learners advance in the instructional program or complete program 
educational requirements that allow them to continue their education or training. 

• Evaluation teams in four programs cited "Continued Education and 
Training" as an indicator of quality in workplace education. 
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• #7. Program successfully recruits the population in the community 
identified in the Adult education Act as needing literacy services. 

• Evaluation teams in five programs cited "Identify and Recruit Program 
Participants" 



• #8. Students remain in the program long enough to meet their educational 

needs 

• Evaluation teams in two cited "Retention of Employees in the Education 



Program" 



In his article "Evaluating National Workplace Literacy Programs," Thomas 
Sticht states that the evaluation of programs funded through the National 
Workplace Literacy Program should be seen in the context of the goals of that 
program: "The ultimate goal of the National Workplace Literacy Program is to 
improve the productivity of the workforce through the improvement of workers' 
literacy abilities. For this reason, after providing convincing evidence that 
improvements have taken place in literacy abilities, the workplace literacy 
provider needs to present convincing evidence that the improvements in literacy 
have led to improvements in job productivity." (p.10) 



We refer to this part of Dr. Sticht's article to make a point. If the goal of 
the National Workplace Literacy Program is "to improve the productivity of the 
workforce through the improvement of workers' literacy abilities," .then indicators 
of quality in workplace education programs - at least those funded through the 
National Workplace Literacy Program - must reflect what productivity means to 
individual programs, unions, companies, and agencies, and to specific 
industries. If the national indicators are to stand as accurate guide-posts of 
program quality in workplace education, as they are refined and expanded in 
years to come, they should reflect the productivity-related and organization- 
impact goals of workplace education programs. 



IV. Evaluation Summaries of Participating Programs 

The program summaries describe the work which the teams 
accomplished during the course of their evaluations. These summaries are 
contained in the full report. 
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V. Conclusions: What We Learned about Quality Outcomes 

Processes, and the Panning and Evaluation Team 
Method in Cycle 5 

We draw the following conclusions from this evaluation: 
Regarding Indicators of Quality for Workplace Education 

• Indicators of quality for workplace education programs should be reviewed 
with critical attention to the organizational context of these programs. 

Regarding the Effectiveness of the Pla nning and Evaluation Team MnriPi 

• Evaluation should be an ongoing, integrated function of the program 
partnership, represented by the planning and evaluation team, and facilitated 
by, not relegated to, an outside evaluator. 

~ The planning and evaluation team model mirrors the continuous 
improvement process of the total quality workplace. 

-- Planning and evaluation teams are different from advisory boards and 
should not be confused with them. 

• The planning and evaluation team addresses and ensures program 
continuation. 

• The planning and evaluation team process is not only an evaluation process, 
but it is also a staff development process which results in unanticipated 
benefits to the program and host organization. 

• Teams which were established before the evaluation, as part of program start- 
up, had a distinct advantage over other teams. 

• Employee participation on the planning and evaluation team is essential and 
should be increased. 

• Each team and each program progresses at its own pace but the evaluation 
focus should remain on generation of useful data. 

• Feedback from the evaluation facilitator is a very important phase in the 
evaluation process. 

• Team members often do not easily distinguish between quality outcomes and 
processes. 

• Creative and generous documentation and measurement is the frontier for the 
planning and evaluation team. 

• The facilitator is an important part of the team. . 
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VI. Recommendations: Improving the Planning and 
Evaluation Team Model a 

Based on the conclusions of this evaluation, we make the following 
recommendations to the MWEI partners. Person(s) who might follow up on th* 
recommendations are identified in parenthesis. 

• Require programs to establish planning and evaluation teams durinq orooram 
start-up. (MWEI Partnership) a H'"a><«» 

• Require the participation of four employees on each planning and evaluation 
team. (MWEI Partnership, MWEI Coordinator, Program Coordinators) 

• Establish how well teams accomplish their planning and evaluation work as 
an essential outcome of an effective workplace education program (MWEI 
Partnership) v 

• Increase the resources available to planning and evaluation teams Thev 
need: 3 

- at least four facilitated meetings per team. 

- a special consultant on creative documentation and measurement to advise 
teams on the best ways to approach documentation and measurement 

- training in the use of the data-base called Statistical Process for the Social 
Sciences (SPSS) for teams who are ready to use it . 

- materials generated from the Cycle 5 evaluation, including the quality 
indicator charts. (MWEI Partnership) 

• Develop a job description for the facilitator to clarify his/her roles and tasks 
(MWEI Coordinator) 

Begin a mentoring program tor new learning provider staff in the MWEI 
(MWEI Partnership) 

Convene teams regularly (once every quarter) for information sharing 
(MWEI Partnership, MWEI Coordinator, Program Coordinators) 

- Continue to build a data base on all programs in the MWEI with a focus on 
how they define their quality outcomes and processes. (MWEI Coordinator) 

Involve learning providers more directly in the administration of their grants 
especially in cases where Private Industry Councils are the technical 
administrators. (MWEI Partnership) 

Continue to articulate the philosophical and practical link between planning 
and evaluation teams as a way to evaluate workplace basic skills programs 
and continuous improvement strategies that guide the production of quality 
products. (Team Members, the MWEI Partnership, the MWEI Coordinator 
Program Coordinators) 
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